Search Results: "wookey"

17 June 2012

Johannes Schauer: port bootstrap build-ordering tool report 2

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de. Diary June 4 I added the first version of basenocycles.ml to git. Given an initial set of cross built packages, it tries to compile as much as possible on the resulting system in multiple rounds. June 5 During June 3, I discovered and error in my program that would only come up when using the Debian Sid package lists as the input: 
Fatal error: exception Assert_failure("common/edosSolver.ml", 610, 11)
On this day, June 5, I wrote a minimal test case for this problem. The same day, Pietro figured out that this is a bug in dose which will be fixed in the next release. Begin writing code to figure out how important a binary package is for the further build process. Try to use Depsolver.edos_install to find out what packages are needed to make debhelper available. Restructure basenocycles.ml, exclude source packages that already have been built, still trouble with already existing binary packages and Cudf.mem_installed, comment stuff better. June 6 I wrote some crude code (only estimate, not correct, fixed later) that would give a rough overview of how often a given binary package is directly required as a build dependency. Debhelper came out as the most needed package. It is architecture:all, so it does not have to be built but it has unfulfilled runtime dependencies. To make those packages available, 13 (actually 11, fixed later) packages have to be compiled on Ubuntu Natty. But those packages all (except gettext) require debhelper itself to be built. The first dependency cycle. This dependency cycle (actually, the 12 cycles) can be broken by either cross compiling those source packages or by making them build without debhelper. One goal of the program is to help decide what the easier option is, but this is not yet implemented. To play around a bit, I created the possibility to specify a list of packages that are additionally to the minimal set of cross compiled packages also cross compiled. I added the 13 packages found above to the list, thus making the binary packages they build available. This made debhelper available in the system. As a result, 1625 out of 3339 source packages can be built with just a minimal build system (priority:essential packages plus build-essential) and debhelper available. The next package that blocks the most other source packages from being built is cdbs. The next nine packages in that list also require cdbs so it seems to be the next important package to be available. Pietro's suggestions make me: 
 - do not open BootstrapCommon but ExtLib, Common, Algo, Debian
 - do proper option parsing and logging
 - use Debcudf.ignore_essential = true
 - do Debcudf.init_tables (binlist@srclist)
 - use @ with shorter list first
 - use more List.rev_append instead of @
 - use CudfAdd.who_provides to find if a package is available
June 7 Pietro and Patrick suggest that for solving the debhelper cycles, one could provide a minimal debhelper version so that the above list of 12 packages can be built without debhelper. I try to figure out how to get a list of packages that are missing to make a package installable/buildable. This functionality should be provided in dose but I fail to find it. June 8 Lacking a solution of the problem of June 7, I write a mail to Pietro. I start my first graphs in ocaml using the ocamlgraph library. The graph I generate, starts off at a binary package. For each binary package it connects new vertices as its runtime dependencies. If a binary package is not arch:all and also not yet otherwise compiled, its source package is also added. The result is a graph in which set of source packages in it will make the initial package available, if those source packages would be cross compiled. The graph is extended further than the source packages. June 9 I refine a couple of functions, make univ_get_pkg_by_name return the package with the highest version number. I wrote a rather lengthy (1027 words) email to the list that explains my status as of this day. I can create graphviz dot files with ocaml, can create node and edge types and create the graph by an imperative pattern that I saw a lot in Pietro's code. Disjunctions are not yet handled correctly (see mail from June 8). The graphs generated look like the following: http://mister-muffin.de/p/8nyc.png June 11 I write a test case which shows how CudfAdd.who_provides doesnt find virtual packages. Automate the process of finding the packages that, if cross compiled, would make another package available. Add more predicates (identifying source packages) and improve input file reading code. Move build_compile_rounds which compiles as many source packages as it can in multiple rounds on a given minimal system a toplevel function and thereby abstract it more. Create a rudimentary text based menu to choose different actions to take for an analysis. Start writing an extended version of simple_dependency_graph for deeper analysis. Use xdot to show graphs from the text menu. Allow saving those graphs to a file. June 12 Move functionality from the extended version of simple_dependency_graph over to the normal version and delete the extended version. Add the new Closure vertex type. Create extended_dependency_graph which is supposed to not contain single binary package vertices but handle a package and its installation set as one vertex. The output of extended_dependency_graph is optionally reduced to the biggest (non degenerate) strongly connected component. User gets the option of choosing the exploration depth. June 13 Pietro replies to my mail from June 8 but apparently I failed to express myself well enough in my last mail, so I rephrase my question. Pietro replies to my email from June 11 and explains how the effect I see is due to "a nuisance of the debian to cudf encoding". As a result I change my code accordingly. June 14 Another lengthy (1130 words) email to the list. I explain what was done in the past days, what parts work and how they work. I list some rationales on why I did things the way I did them. The most important observation is, that after improving my code again and again, I ended up representing the dependency cycle problem in the same (very similar) way that Pietro suggested in the beginning. This is probably a good discovery. Lots of data of that email is now of only little use as of June 16, I make lots of improvements in correctness. As I dont have an answer to my other email to Pietro from June 13, I implement a very crude way to get an answer to the question of what packages are missing for a package to be available/compileable. I called it flatten_vpkgformula_best_effort and it suffers from many faults including disjunctions and package conflicts. Patrick spots a problem. As a result, I make sure that at no point, the source package of an arch:all package can be listed. June 15 As a reply to my mail from June 13, Pietro creates a new branch in the git and adds the code I needed to get a proper installation set. June 16 As a result of Pietro's efforts from June 15, I make great advancements on all fronts. Details of the current status follow in the next section. Results A big leap was made on June 16 due to Pietro's great help on making me understand how Depsolver.listcheck can be used for my purposes. My difficulties in finding the solution myself are rooted in many parts of the dose framework being poorly commented but Pietro did already a couple of documentation commits whenever things were unclear for me. Using Depsolver.listcheck makes it possible to be more distribution agnostic and I dont have to handle vpkgs, virtual packages and constraints myself anymore. The code also doesnt suffer anymore by wrongly analyzed dependencies and conflicts. The only thing that is not yet taken care of, is that Depsolver.listcheck only chooses one out of several possible installation set. A final version should be able to take into account that a different installation set could provide a better solution. Overall, in comparison to two weeks ago, I can now properly build, traverse and analyze graphs, can choose an installation set properly, understand more about dependencies, closures, dose and ocaml in general. Finding the importance of binary packages for building When calculating how many source packages are depending on the availability of a binary package I originally flattened the pkg.Cudf.depends list twice for a rough overview. This is of course wrong due to disjunctions and conflicts and also doesnt provide a deep dependency inspection. The new method is to calculate an installation set that is necessary to compile a source package for every source package. The resulting list of binary packages is then used to find out how often a binary package appears in an installation set. I see three drawbacks though: Removing simple graph The simple graph which contained single binary and source packages was removed. I realized it doesnt really serve any purpose to look at it. As a result, Bin vertices and InstallDep edges are also not part of the graph anymore. Since it was responsible for generating the list of source packages that have to be cross built to make a package available, I created a new function get_cross_source_packages which uses an installation to serve the same purpose. Fix extended_dependency_graph extended_dependency_graph now uses installation sets for generating the list of packages that is needed to compile a source package or install a binary package. The list of build dependencies does not include packages that are already installable. The list of runtime dependencies does not include packages that are otherwise available (cross built, arch:all...). Instead of checking for list membership all the time, I created hash tables for the list of installable as well as for the list of available binary packages. Future There are two big tasks for the next two weeks: Task one is to find a way to give hints on which packages to best consider for having reduced build dependencies. This would then probably finally make use of Pietro's cycle algorithms. Task two is to find a way to break cycles and create a build-DAG from a list of packages that already have staged build dependency information. Patrick is currently working on patching dpkg with Build-Depends-StageN dependencies as making perl cross compilable. If he doesnt need the ability to decide which packages to modify to have staged build dependencies in the near future, then task one is probably less urgent and therefor of less importance right now? On the other hand, I can easily generate fake reduced build dependencies so that doing task two right now would not be a problem. Also, having the solution for task two will make it possible to show the user what effect it would have to add reduced build dependencies to a package. For the reasons above (it's not urgent, task one profits from task two being solved) I will go and implement task two first (if there are no objections from my mentors). Another idea, that I discussed with wookey and Patrick yesterday, was that due to multiarch being used for more and more packages, there should exist a set of packages that is cross compilable without any change to the package. We agreed that I make a list of packages that, if cross compiled, would break dependency cycles and make other packages available. I created such a list of about 160 packages for Ubuntu Natty that, if cross compiled, made it possible to have 87% of Natty available (those numbers have to be treated with caution as I did not yet use the proper way of installation sets when generating that list, but the order of magnitude should be correct). Wookey can then try to cross compile those packages. If some packages of those "crucial" source packages are found to be cross compilable, then they should be cross compiled because it means that no work has to be done to break some cycles. Cross compiling all packages that are cross compilable out of the box is no solution, as only natively compiled packages can go into the archive. This is why the list of potentially additionally cross compiled source packages has to be kept as small as possible.

3 June 2012

Johannes Schauer: port bootstrap build-ordering tool report 1

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de. Diary May 21 May 22 May 23 May 24 May 25 May 29 May 30 May 31 June 1 June 2 Results I learned a good chunk of ocaml and how to use dose3 and libcudf. I created a gitorious project and a git repository for all the sourcecode. 
git clone git://gitorious.org/debian-bootstrap/bootstrap.git
The git as of now contains 30 commits and 1197 lines of ocaml code. So far, 62 emails have been exchanged between me and Pietro and Wookey. I created a mailinglist for this project where all email exchange so far is publicly accessible in the archives. You can also download all of the email exchange in mbox format. Everybody is welcome to join and/or read the list. What seems to be finished: the program that finds the minimal amount of source packages that have to be cross compiled to end up with a minimal build system. What it does is:
  1. get all essential packages
  2. get their runtime dependencies
  3. get build-essential plus runtime dependencies
  4. get all source packages that are necessary to build 1.-3. those are the packages that have to be cross compiled
  5. get a list of all packages that are built by source packages from 4.
  6. add all packages from 1.,2.,3. and 5. plus all arch:all packages to a universe
  7. use Depsolver.trim on that universe to figure out which of those packages are actually installable
The result of 7. will then contain a list of packages that are available automatically on the foreign system due to cross compiled source packages and arch:all packages. For Debian Sid, the output of my program is: 
# (1) number of packages with priority:required: 62
# (2) plus, number of dependencies of priority:required packages: 20
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 71
# number of additional packages built from the above source packages: 292
# (4) number of packages of those plus arch:all packages that are installable: 6421
# total number of installable packages (1)+(2)+(3)+(4): 6534
For Ubuntu Natty it is: 
# (1) number of packages with priority:required: 96
# (2) plus, number of dependencies of priority:required packages: 7
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 87
# number of additional packages built from the above source packages: 217
# (4) number of packages of those plus arch:all packages that are installable: 2102
# total number of installable packages (1)+(2)+(3)+(4): 2236
So for Debian, 71 source packages definitely have to be made cross compilable while for Natty, the number is 87. The last two days I was toying around with these minimal systems to see how big the number of source packages is, that can be built on top of them without running into dependency cycles. After installing the binary packages that were built, I checked again until no new packages could be built. For Natty, I was only able to find 28 additional packages that can be built on top of the 2236 existing ones. This means that a number of dependency cycles prevent building anything else. In the coming two weeks I will focus on coming up with a tool that cleverly helps the user to identify packages that would be useful to have for building more packages (probably determined by how many packages depend on it - debhelper is an obvious candidate). The tool would then show why that crucial package is not available (in case of debhelper because some of its runtime dependencies are not available and require debhelper to be built) and how the situation can best be resolved. The possible methods to do so are to identify a package that is part of a cycle and either cross compile it or let it have staged build dependencies.

Johannes Schauer: cross-compilable and bootstrappable Debian

When packaging software for Debian, there exist two important assumptions:
  1. Compilation is done natively
  2. Potentially all of Debian is available at compile time
Both assumptions make the life of a package maintainer much easier and they do not create any problem unless you are one of the unlucky few who want to run Debian on an architecture that it does not yet exist for. You will then have to use other distributions like OpenEmbedded or Gentoo which you compiled (or retrieved otherwise) for that new architecture to hack a core of Debian source packages until they build a minimal Debian system that you can chroot into and continue natively building the rest of it. But even if you manage to get that far you will continue to be plagued by cyclic build and runtime dependencies. So you start to hack source packages so that they drop some dependencies and you can break enough cycles to advance step by step. The Debian ports page lists 24 ports of Debian, so despite its unpleasant nature, porting it is something that is not done seldom. The process as laid out above has a number of drawbacks: If Debian would provide a set of core packages that are cross-compilable and which suffice for a minimal foreign build system, and if it would also have enough source packages that provide a reduced build dependency set so that all dependency cycles can be broken, building Debian for a yet unknown architecture could be mostly automated. The benefits would be: With three of this year's GSoC projects, this dream seems to come into reach. There is the "Multiarch Cross-Toolchains" project by Thibaut Girka and mentored by Hector Oron and Marcin Juszkiewicz. Cross-compiling toolchains need packages from the foreign architecture to be installed alongside the native libraries. Cross-compiler packages have been available through the emdebian repositories but always were more of a hack. With multiarch, it is now possible to install packages from multiple architectures at once, so that cross-compilation toolchains can be realized in a proper manner and therefor can also enter the main archives. Besides creating multiarch enabled toolchains, he will also be responsible for making them build on the Debian builld system as cross-architecture dependencies are not yet supported. There is also the "Bootstrappable Debian" project by Patrick "P. J." McDermott and mentored by Wookey and Jonathan Austin. He will make a small set of source packages multiarch cross-compilable (using cross-compilers provided by Thibaut Girka) and add a Build-Depends-StageN header to critical packages so that they can be built with reduced build dependencies for breaking dependency cycles. He will also patch tools as necessary to recognize the new control header. And then there is my project: "Port bootstrap build-ordering tool" (Application). It is mentored by Wookey and Pietro Abate. In contrast to the other two, my output will be more on the meta-level as I will not modify any actual Debian package or patch Debian tools with more functionality. Instead the goal of this project is threefold:
  1. find the minimal set of source packages that have to be cross compiled
  2. help the user to find packages that are good candidates for breaking build dependency cycles through added staged build dependencies or by making them cross-compilable
  3. develop a tool that takes the information about packages that can be cross compiled or have staged build dependencies to output an ordering with which packages must be built to go from nothing to a full archive
More on that project in my follow-up post.

25 May 2012

Johannes Schauer: setting up mailman, postfix, lighttpd

I was worried about having to learn hundreds of configuration options to properly set up mailman, postfix and lighttpd on Debian Squeeze. Turned out, that except for lighttpd it all works out of the box. 
apt-get install postfix
When asked by debconf, I specified lists.mister-muffin.de as the fully qualified domain name. 
apt-get install mailman
newlist mailman
The newlist command reminds me that I have to add its output to /etc/aliases. After doing so, I have to run: 
newaliases
From now on, I can add any mailinglist by running newlist, editing /etc/aliases and running newaliases. Mailinglists can also be added through the mailman webinterface but one still has to put the according entries into /etc/aliases. Following is a working lighttpd configuration that works out of the box with the default settings of mailman on Debian squeeze. This was the only part that caused me some headaches. 
server.modules += ("mod_alias", "mod_cgi", "mod_accesslog")
$HTTP["host"] == "lists.mister-muffin.de"   accesslog.filename =
    accesslog.filename = "/var/log/lighttpd/lists-access-log"
    alias.url += (
        "/cgi-bin/mailman/private/" => "/var/lib/mailman/archives/private/",
        "/cgi-bin/mailman/public/" => "/var/lib/mailman/archives/public/",
        "/pipermail/" => "/var/lib/mailman/archives/public/",
        "/cgi-bin/mailman/"=> "/var/lib/mailman/cgi-bin/",
        "/images/mailman/" => "/usr/share/images/mailman/",
    )
    cgi.assign = (
        "/admin" => "",
        "/admindb" => "",
        "/confirm" => "",
        "/create" => "",
        "/edithtml" => "",
        "/listinfo" => "",
        "/options" => "",
        "/private" => "",
        "/rmlist" => "",
        "/roster" => "",
        "/subscribe" => "")
 
server.document-root        = "/var/www"
server.errorlog             = "/var/log/lighttpd/error.log"
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "www-data"
server.groupname            = "www-data"
index-file.names            = ( "index.html" )
server.dir-listing          = "disable"
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
As a bonus, I wanted to import my existing email exchange with my GSoC mentors into the mailinglist. First I was planning on manually sending the email messages to the list, but a much easier option is to just import them in mbox format. To extract all email messages, I first wrote the following python snippet: 
import mailbox, itertools
box = mailbox.mbox('~/out')
for message in itertools.chain(mailbox.mbox('~/sent'), mailbox.Maildir('~/Mail/Web/', factory=None)):
    if (("wookey" in message.get('to', "").lower()
      or "wookey" in message.get('cc', "").lower()
      or "wookey" in message.get('from', "").lower()
      or "abate" in message.get('to', "").lower()
      or "abate" in message.get('cc', "").lower()
      or "abate" in message.get('from', "").lower())
      and not message['subject'][0] == '['
      and not message['subject'] == "multistrap"):
        box.add(message)
box.close()
It iterates through messages in my mbox and maildir mailboxes, filters them for emails by wookey or pietro, strips away some messages I found to not be relevant and then saves the filtered result into the mbox mailbox ~/out. It is important to specify factory=None for the Maildir parser, because it otherwise defaults to rfc822.Message instead of MaildirMessage. Also do not forget to call box.close(). I initially forgot to do so and ended up with missing messages in ~/out. I then copy the archive in its place: 
scp out lists.mister-muffin.de:/var/lib/mailman/archives/private/debian-bootstrap.mbox/debian-bootstrap.mbox
Another thing that initially caused me trouble, was that the mbox didnt have the correct permissions due to the scp. Fixing them: 
chown -R list:www-data /var/lib/mailman/archives/private/
chmod 664 /var/lib/mailman/archives/private/debian-bootstrap.mbox/debian-bootstrap.mbox
And update the mailman archive like this: 
sudo -u list /usr/lib/mailman/bin/arch debian-bootstrap /var/lib/mailman/archives/private/debian-bootstrap.mbox/debian-bootstrap.mbox
Initially I was running the above command as root which screws up permissions as well.

4 March 2012

Stefano Zacchiroli: bits from the DPL for February 2012

Released a few hours ago, here is the monthly report of DPL activities for February 2012.
Howdy, dear Project Members,
here's another round of updates about what has happened in DPL land, this time during February 2012. Highlights Quit a bit of highlights for this month: Talks, interviews, and the like Sprints Plenty of sprints related news! It would be amazing to have an average of one sprint per month for 2012, and we're on good track for it. If you want to help, organize one for your team as documented on the wiki. Legal stuff Appointments In addition to the GSoC admins delegation (see above), I've agreed with former secretary Kurt Roeckx to reappoint him as a secretary for another year. Many thanks, Kurt! Miscellaneous Happy Debian hacking.
PS as usual, the boring day-to-day activity log is available at master:/srv/leader/news/bits-from-the-DPL.*

29 January 2012

Gregor Herrmann: RC bugs 2012/04

good news: from looking through RC bugs in the BTS, it seems that more & more people are starting to join the RCBW effort!

& here's my usual list for the past week:

22 January 2012

Gregor Herrmann: RC bugs 2012/03

here's the list of RC bugs I've worked on during the last week:

9 August 2011

Aurelien Jarno: Debian s390x port (aka 31 bits is not enough)

During Debconf 11, I got access to a fast s390 machine, and I have started to work on a Debian s390x port, the 64-bit version of the s390 port. One of my goal was to help the SPARC64 port, as some of the issues are the same: both are 64-bit big-endian, don t support unaligned access and behave differently between -fpic and -fPIC. Why such a port? When talking about 64-bit ports, we usually hear: 4GB is enough, handling 64-bit takes more memory . This really sounds like 640K ought to be enough for anybody . The s390 port is actually 31-bit from the address point of view (one bit is reserved for address space extension from 24 to 31 bits), so each process is limited to 2GB only. Nowadays applications which need more than 2GB are not that uncommon, especially on mainframes. Actually the 2GB limit already causes some problem in Debian: in some cases it s not possible to build haskell applications or even C applications using GCC. On the other hand, we already require a 64-bit kernel on the s390 port (only the userland is 32-bit), and applications are handling more and more 64-bit or greater values (files offset, time counters, uid, etc.). What is the status? Bootstrapping the architecture was not really easy (as for any other new architectures), due to a huge amount of dependencies and build-dependencies loops, as explained by Wookey during Debconf11. Now that this part is mostly done, an autobuilder has been started and currently more than 65% of the packages are built. The s390x port is hosted on debian-ports.org. Unfortunately it is not yet deboostrapable, though that should happen in the next few days (only a few packages are missing). The main issues are currently packages which fail to build from source due to linker, gcc-4.6 and curl changes, or due to the libjpeg and multiarch transitions, and thus are not directly related to s390x. If your package is in this case, it would be a good idea to fix it. Otherwise if it has a lot of reverse dependencies and the bug is opened for a while, just expect an NMU (as allowed by the 0-day NMU policy). Of course for a few packages s390x specific fixes are needed, some of them are already in the BTS. How you can help? The list of bugs blocking the s390x port is available through the s390x usertag, fixing these bugs (a lot of them are general FTBFS) would help a lot. Alternatively if you have access to an s390x machine you can take a look at the packages failing to build. Update: Fixed the explanation about the 32th bit, thanks to Bastian Blank for the comment.

25 April 2011

Obey Arthur Liu: Welcome to our 2011 Debian Google Summer of Code students!

I d like to extend a warm welcome to our new batch of students selected for the 2011 Debian Google Summer of Code! They should soon be posting on Debian Planet and you re welcome to come talk to them on #debian-soc on irc.debian.org Further details will be posted in the coming days to our wiki: http://wiki.debian.org/gsoc Automated Multi-Arch Cross-Building and Bootstrapping aka autocrossbuild , by Gustavo Prado Alkmim, mentored by Wookey
Enable easy and automated setup of cross-platform automated build systems and bootstrapping for QA in the Multi-Arch era. This involves the creation of multi-stage bootstrap build sequencing tools and a reliable automated multi-arch cross-builder. APT/Dpkg Transaction Ordering for Safety and Performance aka aptordering , by Chris Baines, mentored by Michael Vogt
The ordering code in libapt is responsible for ordering the unpacking/configuration of debs so as to ensure dependencies are satisfied etc. Currently it organizes the ordering into big batches. This project further implements an ordering satisfying more constrains such as minimal amounts of dpkg invocations , minimal amount of broken packages at any point . DebDelta APT Native Integration aka debdelta , by Ishan Jayawardena, mentored by Michael Vogt
Improve user experience of APT and its front-ends by speeding up the upgrade process. This provides a better framework for unified handling of debdelta and future APT improvements such as parallelism. Support for stable and security ugprades as well as multiple APT related libraries is expected. Dpkg Declarative Diversions aka declarativediversions , by Sam Dunne, mentored by Steve Langasek
The dpkg-divert command should be replaced with a new control file with a declarative syntax which Dpkg will parse and process directly as part of the package unpack and removal phases, eliminating the problems resulting from non-atomic handling of diversions. Backend Tools and Infrastructure for DEX aka dextools , by Nathan Handler, mentored by Matt Zimmerman
EX is a new program designed to help improve Debian and its derivatives by merging in changes made downstream and encouraging discussions between the various projects. As this is a new project, most of the infrastructure does not exist (or is rather hackish and incomplete). This project will create the necessary backend tools and infrastructure so that all Debian derivatives can easily make use of the DEX project. Jigsaw Modularized Java in Debian aka jigsaw , by Guillaume Mazoyer, mentored by Tom Marble
The Java Development Kit (JDK) is a big monolithic software tool: many of its features are only useful in limited areas (GUI toolkits are useless for a web server). This project will bring the Jigsaw modular JDK to Debian, helping performance (start-up, size, etc) but also the dependency resolution (to match Debian packaging). Some work exists upstream does not fit with Debian. This project will package the current development version of Jigsaw, update Debian Java Policy, and create the necessary packaging tools for software depending on it. Python Multi-Build for Python Extensions Packaging aka pythonmultibuild , by Mesutcan Kurt, mentored by Piotr O arowski
This project creates a tool to build Python extensions for all Python versions supported by Debian at the time. The project should detect the upstream build system and testing frameworks and use them. It will be interfaced with CDBS and the dh sequencer, replacing their Python snippets. Debian Teams Activity Metrics aka teammetrics , by Sukhbir Singh, mentored by Andreas Tille
This project will gauge the performance of teams in Debian by measuring metrics such as: postings on relevant mailing lists, package upload records from the Ultimate Debian Database and commit statistics from project repositories The information gathered will help in evaluating team performance by measuring how people in a team are working together. An interface to access this information easily will also be developed. Compute Clusters Integration for Debian Development and Building aka computeclusters , by Rudy Godoy, mentored by Steffen M ller
The project s main goal is to enable developers to easily use compute clusters (Eucalyptus, OpenStack ) as environments for arch-specific development by providing a set of tools they can use to setup and run an extended platform for their development, testing and building tasks. Good luck to everyone!

6 March 2011

Neil Williams: Checking build-dependencies

I've been nagged for a while (by Wookey mainly) about the unhelpfulness of dpkg-checkbuilddeps when it outputs the versions and alternatives alongside the package names which are missing, making it harder to pass the list to the package manager. apt-get build-dep isn't particular helpful either - it doesn't look at the modified package at all.

Of course, once the output is in a suitable format, a new script might as well make it possible to simply pass that output to said package manager. Once it can do that, it can then pass the output to the cross-dependency tools, like xapt.

So, I've refactored the embuilddeps script in the xapt package to do just this.

It's gained a few features in the process:

  1. Support for Build-Conflicts resolution

  2. Support for virtual packages, swapping the virtual for the first package to provide it (borrowed some code from Dpkg::Deps for that one).

  3. Support for Build-Depends alternatives (currently using the buildd default of "first alternative gets first chance")

  4. Reads data from debian/control, not the apt cache - to help with the package you're building instead of the one you've already uploaded.

  5. Handles cross dependencies (which are always assumed to not currently be installed) and native dependencies. This support is transitory until such time as enough packages are Multi-Arch compatible that Cross-Multi-Arch becomes trivial.

  6. Support for being used as a build-dependency resolver in pbuilder, including cross-architecture dependencies with pdebuild-cross.

  7. Can locate a debian/control file in a specified directory without needing to be called from that directory

  8. Checks your apt-cache policy to see if the required version of a package is available from your current apt sources. Fails completely if not. (The pdebuild-cross usage will need that to be extended a touch to look at the apt-cache policy from within the chroot.)

  9. Hector Oron has also been asking me to get embuilddeps working with sbuild, so I'm working on that feature too.

  10. verbose and quiet support (so use -q inside other scripts)

  11. most output is already translated - more translations are welcome, especially for the documentation, but hold on until this version has actually been uploaded.



More testing is needed, particularly that the extensive refactoring hasn't broken the pbuilder resolution support and then looking at what still needs to be done for sbuild support. The new script is in Emdebian SVN.

The first-choice method for virtuals and alternatives may well bear extension to explicit management via command-line options. I'm unsure yet whether it needs to be a configuration file setting. It could simply be a recursive - try one, move on - model.

10 April 2009

Obey Arthur Liu: Google Summer of Code 2009: Debian s Shortlist

Copy of http://lists.debian.org/debian-devel/2009/04/msg00421.html. Hi folks, We have been pretty busy these past few weeks with the whole Google Summer of Code 2009 student application process.
I can say that we have this year a very good set of proposals and I d like to thank all the students and mentors for this. I am going to present to you our shortlist of projects that we would like to be funded and believe we can reasonably manage to get funded. As always, remember that the number of slots is not final yet at this point so we can t promise anything. The first preliminary slot count given today was *10* (same as last year) and we hope to get *2* more (as we did last year). This shortlist is alphabetically ordered because we don t want to reveal the current internal rankings. I am inviting you to debate what you think is cool, what is useful, what is important to Debian, maybe give us pointers to resources or people that could be helpful for the projects. We will try to alter our current rankings to reflect the zeitgeist in Debian, while taking into account the personal information that we have about each student involved. The deadline for any modification is on the 15th, so get everything in by the 14th. The final selected projects will be announced by Google April 20th, ~12 noon PDT / 19:00 UTC. We ll have another announcement then. Three proposals need or may need a mentor, I indicated it. For more information about the projects or mentoring and how to talk to us directly, scroll down past the list. Debian s Shortlist : - Aptitude Package Management History Tracking
- Automatic Debug Packages Creation and Handling
- Debbugs Web UI: Amancay Strikes Back
- Control Files Parsing/Editing Library/Qt4-Debconf Qt4-Perl bindings
- Debian-Installer Support for GNU/kFreeBSD
- KDE/Qt4 Adept 3.0 Package Manager
- Large Scientific Dataset Package Management
- MIPS N32 ABI Port
- MTD Embedded Onboard flash Partitioning and Installation
- On-demand Cloud Computing with Amazon EC2 and Eucalyptus Integration
- Port back update-manager to Debian and all Derivatives
- Debian Autobuilding Infrastructure Rewrite And the details: Aptitude Package Management History Tracking Student: Cristian Mauricio Porras Duarte, Mentor: Daniel Burrows Aptitude currently does not track actions that the user has performed beyond a single session of the program. One of the most frequent requests from users is to find out when they made a change to a package, or why a package was changed; we want to store this information and expose it in the UI in convenient locations. As a side effect, this might also provide some ability to revert past changes. Automatic Debug Packages Creation and Handling Student: Emilio Pozuelo Monfort, Mentor: Marc Brockschmidt This proposal aims at providing debug binary packages for the packages in the Debian archive in an automatic manner, moving them away from the official Debian archive to an special one. This has the benefits of providing thousands of debug packages without any work needed from the developers, for all the architectures, without bloating
the archive. Debbugs Web UI: Amancay Strikes Back Student: Diego Escalante Urrelo, Mentor: Margarita Manterola The Amancay project aims to be a new read/write web frontend to Debian s BTS; allowing DDs and contributors to easily interact with bugs via an intuitive yet powerful interface, enabling new workflows and creating new contribution opportunities like triaging while upholding reporting quality. Control Files Parsing/Editing Library/Qt4-Debconf Qt4-Perl bindings Student: Jonathan Yu, Mentor: (probably) Dominique Dumont see below This project proposes a common library for parsing and manipulating Debian Control files, including control, copyright and changelog. Main ideas include validating and parsing of these files, with both Strict and Quirks modes for the parser. The second idea is a new frontend for Debconf using Qt4 (for which Perl bindings will be written). Debian-Installer Support for GNU/kFreeBSD Student: Luca Favatella, Mentor: Aurelien Jarno GNU/kFreeBSD is currently using a hacked version of the FreeBSD installer combined with crosshurd as its own installer. While this works more or less correctly for standard installations (read: the exact same installation as in the documentation), it does not allow any changes in the installation process except the hard disk partitioning. This project is about porting debian-installer on GNU/kFreeBSD, and to a bigger extent, make debian-installer less Linux dependant. KDE/Qt4 Adept 3.0 Package Manager Student: Mateusz Marek, Mentor: NEEDS MENTOR, see below. Finish Adept 3.0, a fully integrated package manager for Qt4/KDE4. Adept is currently the only viable path to a Debian native package manager on KDE that would support modern features such as tags, indexed search or good conflict resolving. With Aptitude-gtk still in development and only available for GTK+ and (K)PackageKit having fundamental problems, Debian needs this project to stay in control of its package management on KDE after much neglect in the recent years. Large Scientific Dataset Package Management Student: Roy Flemming Hvaara, Mentor: Charles Plessy Large public datasets, like databases for bioinformatics are typically too big and too volatile to fit the traditional source/binary packaging scheme of Debian. There are some programs that are distributed in Debian, like blast and emboss, that can index specialised databases, but Debian lacks a tool to install or update the datasets they need and keep their indexing in sync. MIPS N32 ABI Port Student: Sha Liu, Mentor: Anthony Fok This project first focuses on creating a new MIPS N32 ABI port for Debian. Different from O32 and N64, N32 is an address model which has most 64-bit capabilities but using 32-bit data structures to save space and process time. A second focus will be given on making such a mipsn32el arch fully optimized for the Loongson 2F CPU which gains more and more popularity in subnotebooks/netbooks in many countries. MTD Embedded Onboard flash Partitioning and Installation Student: Per Andersson, Mentor: Wookey Many embedded devices have MTD onboard flash as persistent storage like the Kurobox Pro NAS, the Neo Freerunner, the Sheeva Plug or the OLPC. With MTD flash being so popular and with increases in capacity, support for MTD partition/installation would make Debian even more interesting to a wide range of of devices, making it one step closer to being universal. On-demand Cloud Computing with Amazon EC2 and Eucalyptus Integration Student: David Wendt Jr, Mentor: (probably) Steffen Moeller see below In many academic fields, as well as commercial industries, people use clusters to distribute tasks among multiple machines. Many times this is done by packaging a whole operating system disk image, uploading it onto the cluster, and having the cluster run it in a VM. This project intends to make it easier for Debian to distribute prepared disk images templates like they distribute CD images now, for the users to recreate or customise these templates with Debian packages and for administrators to host such clusters with Debian. Port back update-manager to Debian and all Derivatives Student: Stephan Peijnik, Mentor: Michael Vogt The project would involve taking the distribution-(Ubuntu-)specific update-manager code, analyzing it, and creating a package with just its core functionality, decoupling the distribution-specific parts and thus making the core code extensible by distribution-specific add-ons. This in turn would remove the need of porting update-manager to Debian with every upstream release. An additional optional goal would be replacing the synaptics-backend with a python-apt based one. Debian Autobuilding Infrastructure Rewrite Student: Philipp Kern, Mentor: Luk Claes Rewrite the software that currently runs the Debian autobuilding infrastructure in a way that makes it more maintainable and robust. It will use Python as its programming language and PostgreSQL for the database backend. By harmonizing buildds, many build failures can be prevented and wasteful workload on buildd volunteers can be reduced. On mentoring: Petr Rockai, the original developer of Adept has offered help to anyone willing to adopt Adept. Sune Vuorela has offered help for any Qt4 and KDE related issues. *We really need a mentor here*. The student is quite competent but Google dictates that we provide a mentor to handle student management. Dominique Dumont, although not DD, has signaled interest in mentoring this, although it hasn t been confirmed yet. Sune Vuorela has offered to help co-mentor for the Qt4-Debconf and Qt4-Perl bindings part. Steffen Moeller has signaled interest in mentoring this, although it hasn t been formally confirmed yet. Charles Plessy of the Debian Med team will provide help for use cases related issues. Eric Hammond, developer of the original vmbuilder image creation tool and maintainer of a set of Debian and Ubuntu images will provide help for Amazon EC2 and image creation issues. Chris Grzegorczyk from the Eucalyptus team will provide help for Eucalyptus and Eucalyptus/Debian integration issues. Contacting us: Considering the tight schedule, most stuff happens live on IRC: #debian-soc on irc.debian.org You can also consult our wiki page for some additional information:
<http://wiki.debian.org/SummerOfCode2009> We have a mailing-list at:
<http://lists.alioth.debian.org/mailman/listinfo/soc-coordination> Keep this discussion on debian-devel@lists.debian.org while cc-ing soc-coordination@lists.alioth.debian.org. This thread is for debian-devel primarily.

9 August 2008

Neil Williams: Progress at DebConf

First complete day after a full day travelling and I'm just about over the jet lag. Progress so far:

Overall, very good.
(Posting delayed due to network problems.)

It appears that drivel cannot cope with posting to the blog from DebConf.
:-(

15 April 2008

Neil Williams: PS - on keeping interested during the wait

Something I want to add to a post by Slyvian LeGall - motivation does not have to disappear during the wait.
I was lucky, my advocate is project-lead for a Debian sub-project (Emdebian in this case) and I was able to contribute to Emdebian before my DD status was completed. This included uploads to Emdebian that Debian people could use. It included policy decisions, infrastructure development and programming of bespoke utilities.
i.e. my interest was maintained because I had an outlet and an encouraging, supportive, advocate.
When a DD advocates a particular person in NM, there should be some common ground, some experience of working with that person and it should be possible to find a way to maintain a working relationship within that environment.
My experience of the end of NM was one of increasing responsibility, increasing activity, increasing motivation and constant affirmation and assistance from my advocate and others in the sub-project.
So a big thank you to Wookey and debian-embedded in general. It bodes well for the other prospective DD's currently being mentored and encouraged through NM within Emdebian itself.
(Mind you, my NM timescale shows that I didn't wait anywhere near as long as the current delays.)
I fully appreciate that the horrendous delays currently being inflicted on people in NM and upon Debian in general are abominable. If there are ways that existing teams in Debian can adopt and encourage these prospective developers, it can only help in keeping these people interested. It may be a more productive way of helping with the NM problems than just carping on a blog.

25 October 2007

Neil Williams: Emdebian root filesystem installable

Been a bit busy for the last week or so. Apologies to those who were waiting for other things but I've been dedicating my time to Emdebian in preparation for CELF in Linz, Austria which is now less than a week away.
I spent three days as a guest of Wookey and his employer, TobyChurchill, in Cambridge. TCL are sponsoring Emdebian at CELF and also assisted with this work session so that Wookey and I could work together to fix the root filesystem. I'm very grateful to TCL for donating a balloon3 board that I can use to test all the new packages I'm cross building for Emdebian. Various bugs in emdebian-tools have been fixed, allowing the rootfs scripts to be tested and improved.
Other improvements include outline support for machine-specific customisations of the rootfs from within the tools. Support includes per machine-type and per machine-variant combinations and at three points in the rootfs generation process: package selection, post first_stage_install and post_second_stage_install. I hope to create some extra documentation for those who want to go the whole hog and write a new debootstrap suite-script.
The rootfs uses busybox, dpkg, apt and includes support for ntpdate and cdebconf. Current tarballs are about 14Mb to download, installing to about 25Mb. Two main reasons: Emdebian doesn't recognise "Essential:" and Emdebian doesn't install perl.
I have some more testing to do tomorrow but the current Emdebian target repository does have packages sufficient for a usable root filesystem within a chroot. The remaining stage is combining the kernel modules into the rootfs to make a bootable system.
Support for these new features will be in emdebian-tools 0.4.3, due for upload to Emdebian tomorrow, hopefully. All Debian users of emdebian-tools have access to the emdebian repository so the upgrade will be available immediately after the upload.
Watch the debian-embedded list for full details of the new features (as well as the expanded manpages).

19 July 2007

Eddy Petri&#537;or: How to build rrdtool with soft-float for a debian arm machine (which uses hard float)

Being a good Debian user

Notes: the bulk of this article was written on the 17th; updates were added in later; this article is really long, you might want to read it in more sessions

Debian's arm port uses hardfloat. This means that apps like rrdtool (graph function) take ages to run because there is no floating point unit. In order to make this faster you'll want to use soft-float emulation. But this support needs to be present in all depended upon libs, including libc, which is an ABI incompatible change which would break all binaries. I can't use the yet-in-development-and-unofficial armel port since the machine has to be up and running almost 24/7 and I can't afford downtime.

So what to do?

Well, obviously use another soft-float enable libc, like uclibc, in parallel withe the hrad-float enabled libc. Debian Sid contains uclibc development libs, but building the whole toolchain is a pain. There is no real support in the distro itself and building on the arm machine is kind of out of the question. I tried a little in a debootstrapped chroot on the native machine, but this was a pain and I was stuck tring to figure out how to disable whitles and bells for rrdtool since I didn't needed any of the perl, tcl or python support. So I stopped, went to bed and decided I'll continue the next day (the day I wrote the biggest part of this article). The future looked dark, trying to build a softfloat uclibc rrdtool on a system that wasn't prepared for that and from which softfloat was pulled away somewhere during etch's development stage....

Deciding to go with the competition - native building

From link to link I found out that gentoo has arm-softfloat-linux-uclibc port, among others, so I decided to use that on the target machine (a NSLU2) in a chroot. Well I stumbled on the low computing power, when I tryed to emerge sync the tree. I gave up un that and tried to use emerge-websync. It looked better, but I didn't got too far with this either since the machine ended up being overloaded, with figures varying from 3 to 8 for the load average (all of them).

Deciding to go with the competition - cross building

It was time to bring in the big guns and I decided I was going to crossbuild those binaries, probably statically since it meant lower risk for things to break on the target machine.


Since I observed that gentoo has a nice tool called crossdev which actually uses emerge (the equivalent of dpkg+apt in Debian world) to make the cross chain tool and saw that they have some really good documentation and a development version of an arm-softfloat-linux-uclibc, I decided I could .

So I went on and downloaded the latest stage3 tarball for x86 (so I can have a gentoo system to start with), made all the necessary things and created a gentoo chroot. I emerge sync-ed, updated portage, installed crossdev (following the nice instructions on the gentoo embedded site about the subject). This took a while, but it went like a breeze. I made the necessary arangements to be able to cross build packages (setting SYSROOT, making that nice xmerge script).

As my goal was to make an arm-softfloat-linux-uclibc binary for rrdtool I tried directly to xmerge rrdtool (of course, first just pretended -p).

Finding bugs in the packages

First bug

All went fine until I found freetype which for some weird reason detected the i486-pc-linux-gnu-gcc compiler as i486-pc-linux-gnu (observe the missing trailing -gcc). I looked over the logs and determined that the CC_BUILD environment variable was not set. So exported CC_BUILD=i486-pc-linux-gnu-gcc and xmerged the library.

(While trying to fix this issue I entered #gentoo-embedded on freenode and actually met Yuri Vasilev, whom I had met in Edinburgh, at DebConf. Look for the video about Ligusk if you are interested about his project to make a Debian like distribution based on Gentoo. It was nice to meet a friendly "face" - as much you can say that online.)


I reported the bug (already there is a patch, I haven't tested, but looks ok) and the workaround in the gentoo bugzilla... after creating an account, since I never used that BTS before.

Second bug

Next pain was libart_lgpl whose upstream, for some comfortability reason chose to detect the target type sizes (and basically ignoring inttypes.h) with a small program which was built with the cross compiler which later was supposed to be ran on the host. I don't think is necessary to say that i failed, of course, since the binary was an ARM binary, while the host was an x86 machine. Yupee! not.

Ok, two bugs, no softfloat rrdtool yet. I reported this second bug and tried to work around the issue. Since I didn't really used too much a gentoo system, I wanted to know details and what possibilities I had. I got some really nice tips from "solar" (aka Ned Ludd) on #gentoo-embedded. He pointed out that while building a package one can press CTRL+z, do some stuff, like altering the Makefile.in file in the build directory and then fg back. Ok, cool.

All I need now is somebody with direct access to an arm machine that can compile and run for me the program in question. seanius on #debian-devel was kind enough to do this for me and, to my surprize, I got the same output header file as I got in the host x86 chroot. But that detail is not important, let's just compile that rrdtool. Well, first libart_lgpl...

Ok, I CTRL+Z-ed the build process just after the configure stage ended, cd-ed into the build directory, I tampered with the Makefile.in file and changed that line from

./gen_art_config > gen_art_config.h

into

cp /gen_art_config.h.arm gen_art_config.h

Haha! In your face! Ok, nasty did was done so I fg-ed and went on confident that I will get my hands on that rrdtool very soon. The package build ended succesfully, without any other suprizes, so I went for the graal!


xmerging rrdtool, the bugs don't stop

xmerge -v rrdtool

Bla, bla, bla.... bla, bla, bla, what do you know, there are a few IEEE math function tests in the configuration script of rrdtool. They failed. I don't know why, but I have a feeling that this should be happening on softfloat or even maybe the rrdtool source is not that cross compile aware. I didn't care that much, so I did what I knew best, tampering with an ongoing build.

This time is was a little bit harder to catch since I had to wait for the source to unpack (well, there is a larger frame, but that is the rough timing) and at that moment CTRL+Z the build.

I didn't had much time to waste, so after looking shortly over the tests I realized I should just ignore the error and look for the "failing command" that was interrupting my beloved build. Muhahah!

Thank God for verbose error messages, I could find the place fast and chaged the two "exit 1" statements into two harmless 'echo "Ignoring IEEE math errors"', sic. I had to do this a few times since I got the timping wrong, at some point I got the syntax wrong (at which point I remembered the cp trick I did earlier and saved the modified file in /configure.rrdtool )...

xmerging rrdtool and the 4th bug

After passing past this stage, I got a compilation error which, judging from the command, meant that the linker tried to link the target (arm) binary against the host (x86) libraries.

Oooook, this is not going to stop me! I look at the Makefile.in to try understand why this happened and then realized it wasn't worth it for me to try to fix it the right way. So I cheated once again! Muhaha, I just ran some useful commands which I'll let you feast your eyes to:

ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libart_lgpl_2.so /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libpng12.a /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libfreetype.so /usr/lib/

Ok, good to go. Of course, I had to stop again the build to copy the "fixed" configure script that ignored IEEE math errors, but that is already history.

And, believe it or not, actually the build succeded. I had a cross built rrdtool using shared libraries built against softfloat ucllibc. No guarantees about its correctness, but it built!

This is when I decided to start writing this article.

Still there are things to do

I will have to do a few things before being happy about the victory:

26 June 2007

Jonathan McDowell: Nice guy EDI

I've spent the last week and a bit in Edinburgh, at DebConf7. I found it took me a couple of days to adjust for some reason, but after I'd managed to lose all sense of time everything went much better. I'll probably spend the next week recovering and catching up with real life. I have to say I don't feel I achieved a whole lot while I was there. When I went to my first DebConf (DebConf4, in Brazil) I spent a lot of time hacking and making progress on things. I didn't manage to do so well in Finland, nor this year. I'm not sure that's entirely what DebConf is about though. I saw a lot of people; the UK crowd who turn up to most things, people I hadn't seen since Finland, and those who I hadn't see since Brazil (and must try harder to see more often). And of course several names I was able to put to faces. This is a really useful part of such events - I find it's much easier to deal with people online once I've met them in real life and have an idea where they're coming from. Having said I didn't achieve much I did many to leave with a lot of things in my personal todo list. I had the calendar conversation a couple of times again, at greater length, resulting in being told that I should just do it. So I need to think about a proper design. I started looking at Just Writing Code, but the problem is such that I believe a more rigorous approach is required. Another item was ARM hacking; I sat in on the ARM BOF and caught the end of the Emdebian stuff, as well as talking to Wookey and Neil Williams quite a bit about it. Wookey's convinced me that working on mainline Balloon2 support is still useful, even if everyone is more interested in Balloon3. And I thought a bit about starting to do some AM work again, assuming I can find enough time to do a decent job of it. Mao was sadly lacking; I got a couple of games in, but not the number I'd come to expect from previous DebConfs. I think the trick is to find Marga sooner... In addition to the travel to Edinburgh we ended up with a daytrip to the Isle of Bute - thanks to Patty Langasek for organising this. We managed not to lose anyone, and had a visit to a public loo from 1898. Post DebConf a bunch of us ended up in Solva for a party to celebrate Daniel and Rob's civil partnership. It was well worth the travelling, and good to see people who hadn't been at DebConf. Finally, on the way back home, we detoured via Stonehenge. Anyway. Now I'm back home, Katherine's turned up for a bit and hopefully I'll manage to do some of my todo list. See y'all next year in Argentina, assuming I can find the funds.

20 June 2007

Neil Williams: dpkg filtering

This started as an Emdebian idea but it is growing into a more all-purpose option. The requirement is to be able to filter packages during installation or cross-building so that manpages, infopages, documentation (including changelogs, copyright and readme etc.) and translation files for unsupported locales are omitted. When installing, the files would simply not appear on the filesystem or in the dpkg status information. When building, the files would disappear from the package completely (hence the role of filtering in Emdebian). The intention is that embedded systems can have empty /usr/share/man/ /usr/share/info/ and /usr/share/doc directories and a much reduced number of translation files in /usr/share/locale - the default would be only the translation files for languages that are supported on that machine, via 'dpkg-reconfigure locales'. Initially, only the narrow filters needed by Emdebian were considered - discussions at DebConf7 have shown how this can be easily extended to support arbitrary filters - necessarily the risks of using such optional filters would be made clear to the users in the manpage etc. - so that any file in any package could be omitted from the install process and the package file lists as well as during the preparation of packages for embedded devices.

Filters would be glob patterns held in a file in /etc/ (possibly /etc/dpkg/filters.d/) and would include the option to exclude files en-masse and specifically include some files that may otherwise be excluded in specific circumstances. The same filters could be used when building packages for embedded usage as well as installing Debian packages on systems where space is a problem. The main benefit of this filtering for Emdebian is that it makes it much easier to isolate the package changes needed to make the package cross-compilable from the changes needed to remove stuff that embedded devices don't need but which Debian Policy requires. This in turn makes Emdebian patches more useful to Debian maintainers when bugs are filed on their packages about a failed cross-build.

So far, it's taken up most of my hacking time in DebConf. Ian Jackson, Enrico Zini and Wookey have all helped improve the original ideas during that time. Prototype code is mostly complete but the extended filtering support will now have to wait until after DebConf. I'm hopeful that the patch will be accepted, once it is ready.

If this seems strange or unnecessary, just compare the disc usage of /usr/share/doc (which you can control) with /usr/share/locale/ (which you cannot currently control) or view the baobab images from my current laptop installation.

28 March 2007

Neil Williams: Cross-building a rootfs

With updated cross-building toolchains and a much improved emdebian-tools package, work has finally started on actually building cross-built packages for Emdebian. Wookey has identified the first set - a usable rootfs. http://wiki.debian.org/EmdebianRootfs

Nineteen packages have been cross-built so far, from base-files and popt to util-linux and ncurses so I thought I'd make a note of how it has gone at this early stage. Packages, patches and build logs can be viewed at: http://www.emdebian.org/packages/search.php

Results:
======

1. CDBS cross-builds like a dream. No failures, no missing files, no misplaced files, no files Emdebian doesn't want (like changelogs and manpages). em_make replaces one makefile with a customised one and everything else falls into place. Fantastic.

2. debhelper packages generally cross-build quite well - as long as debian/rules sticks to actually using debhelper rather than creating bespoke rules that just do what dh_foo would have done anyway! (I then have to go behind them and remove the custom rules to get rid of manpages etc. when if the task had been left to dh_foo, em_make would have removed dh_foo automatically. Grrr.)

3. Packages built without CDBS or debhelper are an absolute PITA. I know it is a good exercise to build without "high-level" tools, but PLEASE, only put non-debhelper packages in the main archive if you really cannot have debhelper involved. It'd make my life SO much easier right now. :-)

4. Splitting out the translation files into individual, dedicated, packages one per language is having two related effects: there are a plethora of packages in the repository with only 19 source packages built; the size of the main binary packages has dropped substantially. Some translation packages are 30-60kb in size but in Debian, 12 maybe 30 of these are packaged inside the main binary or a similar combined _common package. Users get no choice, it's no translations or all translations. I cribbed the idea from OpenEmbedded and it's nice to see it paying off, even if it means that 'apt-cache search' listings will be at least an order of magnitude longer in Emdebian. (Sounds like a wrapper is needed.) Need to start testing 'langupdate' soon to make sure it can cope with keeping users up to date with translations of installed programs.

So far, haven't needed too many cache files to be used with ./configure --cache-file but I'm bound to meet those soon.

Overall, things are going well (famous last words).

7 February 2007

Christian Perrier: fr: 99.817%, cs: 92.667%

These are the current translation ratios for French and Czech in unstable, after 20 days of "blitz l10n NMU". No blog entry yesterday, sorry for those of you who like the "bubulle fights the dragon" adventures. I was travelling from Paris to Badajoz/Spain for the Free Software World Conference. Frans Pop, Wookey, Martin Michlmayr, Knut Yrvin and I will give a common talk about the results of the "Extremadura work sessions" funded by the regional government of Extremadura last year. My net connection will be jerky as we have a very good connection at the conference site, but nothing at the hotel (except a ridiculously expensive Telefonia wireless, which can't even be circumvented with IP-over-DNS). viewvc is to be uploaded by ender very soon. No news for dpkg-cross. A NMU is pretty likely now. I just uploaded a few more NMU's today as I got network again:

1 February 2007

Christian Perrier: Solutions Linux 2007

(update: numerous typos removed) This year, attending the whole Solutions Linux event wasn't compatible with my paid work (for those who care, I'm since September head of the Personal and departmental computing unit in the Networks and Information Department of ONERA, the French Aerospace Lab). Despite this, I booked at least yesterday afternoon to visit the expo and more particularly the Debian booth. It is always a pleasure to meet again French developers and contributors as well as all these old dinosaurs which I learned electronic communications with 1200bps modems with, some time ago. The Debian booth was as crowded as usual. It was still featuring a Babelbox, built in one night by Pierre Habouzit and Julien Blache. Congratulations, you did an awful job here. I'm glad I saw all those good people. I was tempted to cite everybody and will finally refrain because I'm too afraid of forgetting one of you guys. For our international readers, this includes several of the people I may have had a few flamy discussions and even blog "wars" with in the recent past (I'm just missing you, Joss, being told that you couldn't attend that Wednesday). As usual, we all survived very well. The other "big" event was also the first formal meeting of the Debian France non-profit organisation. Everything is pretty young but the hidden creation work done by the founding members has been huge and the base is now here. Debian now officially exists in France. I'll do my best to play my part in this action and work with other members of the board to fit the organisation goals and bring Debian more prevalence in our country. I also had a very long talk with Bruno Cornec and Louis Bouchard from HP. We discussed about HP customers needs for official support for Debian on HP machines, and not only servers (ONERA has an internal project to deploy a few hundreds of workstations aimed at being the scientific platform for ONERA scientists and PhD students. We already have more than 100 of these, based on RHEL and guess what are my mid-term plans for them? :-)). We just need some official commitment from HP to support using Debian to unlock the internal fears. It sounds like HP could go this way if customers show enough interest. Of course, ONERA's project is pretty small but we are one of the key players in out domain in the country so that could be kind of interesting marketing for HP...and Debian..:-). Bdale, I need to talk with you soon.... Back to work now, for a few days. Then, I'll take 3 days off next week to attend the 3rd Free Software World Conference in Badajoz and present the results of the Extremadura i18n session we held in September, along with other people who held some of these sessions, namely Frans Pop (about the D-I session), Wookey (about the Embedded Debian session), Knut Yrvin (about the Debian-Edu session) and Martin Michlmayr (about the QA session). All people who I will be very glad to meet again.

Next.

Previous.